An ADMM Solution to the Sparse Coding Problem

For our project, we apply the method of the alternating direction of multipliers and sequential convex optimization to sparse coding of images. The motivation behind sparse coding of images is to model how the brain is able to efficiently utilize the human visual system for a variety of tasks, such as separating a car from a background, as well as general classification tasks. Sparse coding aims to determine a generalized set of overcomplete bases Φ to represent any natural image, where we desire that each image can be represented by a linear combination of a few of these bases, i.e. represented by a sparse vector of coefficients α. When learned on natural images, the sparse coding bases are localized edge detectors in space, orientation and frequency. Feature learning on natural images with sparse coding can be challenging because there can arbitrarily large input dimensions, dependent on how big the visual field for the algorithm is. Further, in many situations, an over-complete set of bases is desired [1], which means that the number of features to learn exceeds the number of input dimensions. Both aspects demands faster algorithms to make the sparse coding problem feasible, either on a single computer or on multiple in a parallel computing setting. Among models for natural images, sparse coding is not a convex objective but is convex when fixing either Φ or α. This formulation makes it slow to train especially when the input dimensions are high. We wish to consider a distributed solution using ADMM, which we believe will speed up the algorithm. Additionally we wish to try out a heuristic in order to be able to use the desired optimization methods, which we will soon discuss.